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Employers from a variety of job placement settings 
and disciplines were surveyed concerning the critical skills for the 
research and evaluation professional, implications for training, and 
future job prospects. The survey, conducted by Gallaudet University 
(Washington, D.C.), also examined attitudes toward hiring hearing 
impaired persons. Through a phone survey of 69 employers of 
evaluation research personnel and a mail survey of a subset of 48 of 
those employers, information was obtained regarding the nature of the 
organization, the kinds of skills needed in research and evaluation, 
the respondent's background, interest in providing an internship, and 
future job prospects. The organizations surveyed represented a range 
of employing organizations. The results of the survey confirmed that 
it is possible to identify a generic set of skills needed to do 
research and evaluation that apply across a variety of types of 
organizations. These include skills in traditional research 
methodology, evaluation, administration and communication, 
statistical analysis, and computer usage. In some settings, 
particular skills would be emphasized more than in others. The 
results indicated a very receptive market for employees or interns in 
large organizations. Positive attitudes toward hiring hearing 
impaired personnel were reported. (The initial questionnaire and 
interview questions are appended.) (MOE) 
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The Employer's view of Research And Evaluation skills 

Donna M. Mertens 
Gallaudet University 
800 Florida Ave,, N.E. 
Washington D.C. 20002 
202 651-5202 

What skills should be included in a research and evaluation 
training program? What other factors need to be considered in a 
research and evaluation program, such as internship experiences? 
What are the job prospects for individuals trained in research 
and evaluation? The answers to these questions depend on the 
source and scope of the information base. A variety of sources 
have been topped in an attempt to give teachers in this field 
guidance in vhe design of their programs. Numerous lists of 
competencies have been published that were based on the logical 
analysis of research and evaluation tasks, as well as on the 
experiences and informed opinions of the early leaders in the 
field of evaluation (Payne, 1974; Sanders, 1979; Scriven, 1974; 
Stufflebeam et al., 1971; Worthen, 1978). 

A second approach to determining what should be taught in 
research and evaluation training programs is to examine the 
course syllabi that are used in such coursework. While 
evaluation training programs tend to be in schools of education 
or psychology, they can also be found in specific disciplines 
such as social work, health, and management settings (May, 
Fleischer, Scheirer, & Cox, 1986). Davis (1986a) edited a volume 
that contains an analysis of evaluation course syllabi from four 
disciplinary perspectives-education, psychology, health, and 
business and management-as well as from an interdisciplinary 
point of view. She reported that specific disciplines differ in 
their curricular emphasis, however, there is increasing agreement 
about the general topics evaluation courses should address 
(Davis, 1986b) . Much valuable information can be obtained from 
this approach. However, additional information is necessary to 
determine the correspondence between curriculxim content and the 
job market, as well as to identify variables related to 
internships and future job placement prospects. 
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Several researchers have investigated the skills that 
evaluators and their employers report as being necessary for 
successful conduct of research and evaluation studies. Two 
studies used this method to examine skills needed within the 
context of educational evaluations. Maryak, Gray, Mehrens, & 
Lezotte (1979) surveyed members of the American Educational 
Research Association's Division H who were primarily employed in 
educational settings. Worthen (1975) interviewed leaders of 
educational agencies that employ evaluators. Anderson and Ball 
(1979) surveyed a group of experts in evaliiation from the fields 
of education, health and social-action programs who were asked to 
rate the importance of a list of conte.it areas and skills. 
Anderson and Ball reported their overall -atings, but they did 
not break down their results to reflect differences of opinion by 
the respondent's field of expertise. 

The present study was undertaken to determine the views of 
employers from a variety of job placement settings and 
disciplinary areas concerning the critical skills for the 
research and evaluation profession, implications for training, 
and future job prospects. Several assumptions guided the design 
of this study. First, employers of researchers and evaluators 
have a valuable contribution to make in terms of identifying the 
skills that are needed by practicing professionals in the field. 
Second, a core of skills exists across disciplines, with 
particular emphasis within specific disciplines. Thus, the views 
of employers should be examined within the context of the type of 
setting in which the evaluation and research occurs. Third, 
evaluators use the tools of research along with supplementary 
skills that are needed to fulfill the special requirements of 
evaluation work. Fourth, individuals who are trained in 
evaluation are often called upon to function in the role of 
researcher, depending on the changing demands of the work 
environment. And, fifth, evaluation courses; can be taught from 
an interdisciplinary perspective (See Conner, 1986) . 

In the current study, the Department of Educational 
Foundations and Research at Gallaudet University conducted a 
market research study that was used as a basis for the 
development of a degree program in research and evaluation that 
is consistent with the existing job market. The general approach 
of the study was to survey potential employers of the program's 
graduates to determine current staffing patterns, skills needed, 
potential internship placements, and anticipated job openings. 
Will the results be of limited value because of Gallaudet 's 
mission to serve the hearing impaired population? Three factors 
contribute to the general izability of the findings. First, with 
the increased implementation of mainstreaming in colleges and 
universities, it is increasingly likely that programs throughout 
the country will be serving students with some type of 
handicapping condition, such as hearing impairment. Second, one 
issue addressed in this study was the conduct of research and 
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evaluation with hearing impaired subjects, which would have 
parallel implications for other researchers and evaluators who 
are being trained to work with special needs populations. Third, 
employers were asked about researchers and evaluators in general, 
as well as particularly about hearing impaired individuals. 

METHODOLOGY 

Subjects . The subjects in the study were potential employers 
of graduates of a research and evaluation program who were 
selected from five categories: consulting firms (including 
private corporations) , government agencies, professional 
associations, residential schools for the deaf, and other 
educational agencies. Large residential schools for the deaf 
were added to the list because Gallaudet serves both hearing and 
hearing impaired students. The initial phase of the study 
involved identifying those agencies that actually employed 
evaluation research personn^sl to serve as the sample. Except for 
the residential schools, the search focused on opportunities in 
the greater Washington D.C. area. 

Organizations that possibly employ evaluation research 
personnel were identified by using the Evaluation Network's 
vacancy list and list of organizations that employ evaluation 
researchers; the C&P Yellow Pages listing of management 
consulting firms; the listing of professional associations in the 

freater Washington D.C. area; directors of educational research 
n the school systems in Washington D.C. and in surrounding 
counties; State Department of Education representatives in MD and 
VA; Gallaudet 's Center for Assessment and Demographic Studies' 
listing of large residential schools; and a listing of federal 
agencies that use evaluations. A computerized database was 
developed of the 288 organizations that potentially employ 
evaluation research personnel. 

In order to identify organizations that actually do hire 
evaluation research personnel, a short questionnaire (see 
Gallaudet College Response Card in the appendix) was sent in the 
Spring of 1986 to these organizations. A follow-up mailing was 
sent to organizations that did not respond to the first mailing 
within three weeks. The purpose of the first questionnaire was 
to determine if the organizations hired evaluators or 
researchers, and if so, if they would be willing to participate 
in a follow-up interview. 

The results of this first survey are presented in Table 1. 
From the initial list of 288 names, a final sample was obtained 
of 81 (28 percent) organizations that hire evaluation research 
personnel and who indicated a willingness to participate in the 
follow-up interview. The other 207 names can be accounted for as 
being undeliverable (11 percent) , not hiring evaluation research 
personnel (12 percent), or not responding to the mailing (49 
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percent) . A 20 percent random sample of nonrespondents was 
contacted by telephone. The results of this follow-up are 
presented in Table 2. The nonrespondents are primarily from 
consulting firms and professional associations (76 percent) . 
Sixty-eight percent of the nonrespondents did not return the 
questionnaire because they felt it did not apply to them. Given 
this information, the large nonresponse rate appears to be due 
primarily to inaccurately identifying the population, rather than 
to a systematic bias in the nonrespondents. 

TABLE 1 



FIRST SURVEY OF POTENTIAL EMPLOYERS 
OF EVALUATION RESEARCH PERSONNEL 



TYPE OF 


ORIGINAL 


UNDELIVER- 


DOES NOT 


NO 


ORGANIZATION 


SAMPLE 


ABLE 


APPLY 


RESPONSE 


Consulting Firm 


96 


29 


4 


50 


Government Agency 


19 


0 


1 


10 


Residential School 


61 


0 


7 


22 


Other Educational Setting 


13 


0 


0 


3 


Professional Association 


99 


2 


22 


57 


TOTAL 


288 


31 


34 


142 



TABLE 2 

FOLLOW-UP OF NONRESPONDENTS TO THE FIRST SURVEY 



TYPE OF ORIGINAL SAMPLE DOES NOT APPLIES APPLIES 



ORGANIZ\TION 
Consulting Firm 


GROUP 
50 


CHOSEN 
7 


APPLY 
4 


BUT 
REFUSED 
2 


AND 

INTERV. 
1 


Government Agency 


10 


1 


1 


0 


0 


Residential School 


22 


6 


3 


1 


2 


Other Educational 
Settings 


3 


0 


0 


0 


0 


Professional 
Associations 


57 


14 


11 


3 


0 


TOTAL 


141 


28 


19 


6 


3 
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TABLE 3 

TYPE OF ORGANIZATIONS INCLUDED IN THE SAMPLE 
OF EMPLOYERS OF EVALUATION RESEARCH PERSONNEL 



TYPE OF 
ORGTINI 7 ATTON 


ORIGINAL 

C AMDT V 
Oxiflxrljri 


REFUSES 
INTERVIEW 


ACTUAL 
INTERVIEW 


SECOND 
QUEST- 
lONAIRE 
RETURNED 


Consulting Firms 


13 


3 


10 


3 


Government Agencies 


8 


1 


7 


6 


Residential Schools 


32 


2 


30 


20 


Other Educational Settings 


10 


2 


9 


5 


Professional Associations 


18 


5 


13 


5 


Unknown 


0 


0 


0 


9 


TOTAL 


81 


12 


69 


48 



The types of organizations in the sample are displayed in 
Table 3. Residential schools for the deaf were represented most 
frequently (40 percent) , followed by professional associations 
(22 percent), consulting firms (16 percent), other educational 
agencies (12 percent), and government agencies (8 percent), (The 
other educational agencies included research departments in 
public schools. State Departments of Education, and university- 
based research centerf.) Twelve of the 81 sample members were 
later unavailable to be interviewed. Therefore, the actual 
rt^sponse rate for the telephone interviews was 85 percent (69 
respondents) . The individuals who were interviewed were asked to 
return a f-^rm indicating their ratings of skills needed in 
evaluation research. Forty-eight individuals returned this 
second form for a response rate of 59 percent on the skill rating 
form. 

The majority of the respondents (62 percent) in the 
telephone interview were trained in research and/or evaluation 
methods. The other respondents were either trained in 
administration/personnel (13 percent), special/deaf education (9 
percent) , or in particular disciplines such as law, medicine, 
history, or English. Seventy-one percent of the respondents 
reported th?t they were actually doing research or evaluation 
themselves. These active researchers/evaluators reported an 
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average of 12.7 years of experience in the field. 



Procedures . The procedures for the identification of the 
sample, development of the database, and the initial survey of 
the potential employers of evaluation research personnel was 
described in the previous section. This section includes a 
description of the procedures used to obtain information from 
those individuals who confirmed that they did employ evaluation 
research personnel and were willing to participate in a follow-up 
interview. 

Two questionnaires were developed: one for use in the 
telephone interview and one for rating possible skills needed in 
the evaluation research profession (see the appendix) . The 
telephone interview questionnaire was designed to collect the 
following information: 

-The nature of the organization 

-The kind of programs/activities occurring there 

-The kind of work that research/evaluation people do 

-Specific examples of evaluation/research projects 

-The kinds of skills needed 

-Whether or not it would be important to be a content 

specialist, and, if so, in what areas 
-Any additional skills that might be needed 

-The respondent's background 

-Whether or not the respondent was trained in research 

or evaluation, and, if so, the nature of that training 
-Whether or not the respondent was actively doing 

research or evaluation, and, if so, for how many years 

-Interest in participating as an internship site 
-Interest in having an intern 
-Process of establishing an internship 
-Interest in having a hearing impaired intern 
-Need for an interpreter 

-Future job prospects 

-Job prospects at their organization 
-Job prospects in the field in general 

The telephone interview questionnaire was pilot tested with 
three organizations similar to those in the sample. The 
interviews were conducted by three trained interviewers during 
May and June 1386. Each interview took about 15 to 20 minutes to 
complete. 

Based on evaluation's reliance on traditional research 
methodologies and the breadth of applications in research and 
evaluation skills, a broad view was adopted of the potential 
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career paths for the program's graduates. Therefore, the 
following description was formulated of the program's intent: 

We are planning a program to train people to 
be evaluators or researchers who can go into 
an organization and help them clarify their 
goals and determine how well they are meeting 
those goals. They will be trained to use the 
scientific method to collect and analyze 
data. These skills could be applied in a 
variety of settings such as in schools to 
evaluate an individual's performance as well 
as at the program level. They can also be 
applied in market research, in the 
establishment of new programs or in projects 
involving survey research. (Taken from the 
Telephone Interview Questionnaire; see the 
appendix. ) 

The skill rating questionnaire was developed based on the 
following sources that were identified in a computerized search 
of the literature related to evaluation research skills: 
Anderson and Ball (1979) surveyed a group of experts in 
evaluation who were asked to rata the importance of a list of 
content areas and skills. Fienberg (1980) recommended a focus on 
learning to use the scientific method for the design of studies 
and the collection and analysis of data to assess the extent to 
which programs or activities met specified objectives. Daudistel 
and Hedderson (1984) recommended a list of topics based on their 
evaluation program in sociology. Wortman, Coraray, and Reis 
(1980) based their list of skills on their program at 
Northwestern University. Maryak, Gray, Mehrens, and Lezotte 
(1979) surveyed members of the American Educational Research 
Association's Division H who were primarily employed in 
educational settings. 

The skill rating questionnaire was sent to the same 
individuals who were later contacted in the telephone interviews. 
These individuals were reminded during the telephone interview to 
return the rating form. They were also asked to identify that it 
came from their organization, if they wanted to. Nine of the 
skill rating forms were returned without such identification, 
consequently, they appear in Table 3 as "Unknown" in reference to 
the type of organization. 

RESULTS 

Nature of the Organization 

A rich picture of the organizations emerged by asking each 
respondent to describe the organization's activities, the kind of 
work that evaluators/ researchers did there, and specific examples 
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TABLE 4i FREQUENCIES FOR EVALUATION/RESEARCH SKILLS 



9A 



Essential De 




NOX 


NO 


RESEARCH METHODS 






Important 


Ansvc 


Sampling 


27 


Id 


2 


3 


Survey research 


31 


± ± 


3 


3 


Questionnaire construction 


31 


1 s 


Ok 

la 


2 


Interviewing skills 


17 


aO 


4 

1 


2 


Content analysis 


19 


AX 


4 


3 


Case studies 


A 




5 


4 


Task analysis 


16 


AA 


b 


4 


Experimental design 


20 


19 


7 


2 


Quasi "*exDerimenta I d^siotn 


1 Q 


17 


7 


5 


Time series design 


5 




14 


4 


Observational research 


13 


A J 


10 


2 


EVALUATION 










Altez native models for evaluation 




1 ^ 

JL D 


1 


3 


Setting goals 


25 




1 


2 


Cost "benefit analysis 


15 


AA 


o 
a 


3 


Grant smanship 


13 


A^ 


1 Ok 

110 


4 


Standards for evaluation 










(ethical/legal) 


29 


14 




«5 

A 


Utilization of evaluation results 


36 


o 


1 

X 




Planning evaluations 


32 


X A 


A 


A 


Assessment of program 










implementation 


27 


1 id 


A 


D 


Heta-evaluation 


2 


29 


7 


10 


Politics of evaluation 


12 


24 


9 


3 


Policy analysis 


14 


A^ 


«d 


2 


Needs assessment 


30 


1 s 

X ^ 


1 

X 


A 


ADHINISTRATION/COHMUNJCATION 










Proposal writing 


23 


AA 




A 


Expository skills (writing 










and speaking) 


w^ 


X A 




4 

1 


Public relations skills 


24 


1 A 
XO 


4 


«5 

2 


Management skills 


28 


la 


1 


1 


Technical writing of reports 


31 


16 


0 


1 


Listening skills 


30 


XO 




X 


Consultations skil Is 


26 


20 


1 

X 


1 

X 


ST ATI ST ICS/COM P UTER 










Data preparation 


27 


1 A 

X a 


01 




Construction of dcita bases 


23 


1 A 
X o 


'a 




Descriptive statistics 


31 


X A 


«d 




Inferential statistics 


22 


A X 


A 


J 


Correlation and regression 


20 


22 

A A 


4 


A 


Multivariate statistics 


16 


24 


A 


A 


Nonparametric statistics 


17 


21 


5 


5 


Quality control 


23 


16 






Elementary computer programming 


12 


25 


9 


2 


Advanced computer programming 


2 


21 


23 


2 


Use of canned programs 


20 


21 


5 


2 


Use of micro computer software 


16 


2(2 


2 


2 
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PSYCHOHETRICS 

T«flt conatructlon 

Reliability 

Validity 

Application of testa (paper and 
pencil ; situational ; performance ) 
Norm and criterlon-ref ernced 
testa 

Selecting a measurement 

instrument 
Assessing a measurement 

instrument 
Interpreting test results 
Reactive concerns and 

unobtrusive measures 



9B 

16 19 a 5 
26 14 3 5 
26 14 3 5 

17 18 a 5 

la 17 a 5 

23 14 6 5 

22 15 6 5 

30 7 6 5 

10 25 7 6 
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evaluations, and surveys of parents and the deaf community. The 
work that was mentioned by these respondents matched that 
described for the other types of organizations, although the 
emphasis was clearly on assessment at the individual level. 

In the other educational agencies, the respondents 
represented offices of testing, research and evaluation in public 
schools and on university campuses. The kind of work mirrored 
that reported in thb other organizations, with a greater emphasis 
on program evaluation. Studies were conducted in such areas as 
prevention of high school drop outs, training school 
administrators to design evaluations , measuring minority 
achievement, building databases of demographic and academic 
information at the local schools, developing consistent reporting 
methods , evaluating the gifted and talented program, and 
conducting reseaich on child development, education and mental 
health in the area of deafness. 

Skills Net>d ed 

Like Anderson and Ball (1979), two of the respondents had 
some difficulty with completing the skill rating form. One 
indicated that skills are important in particular situations. 
Some of the skills are needed sometimes, but not all the time. 
However, when you do need the skill, it is critical. Another 
respondent was uncomfortable with the conceptual inequality of 
the items on the rating form. He felt that some of the items 
represented full courses while others were more individual 
topics. These caveats should be recognized in the interpretation 
of the ratiu^s. 

The respondents' ratings of the skills needed for research 
and evaluation are displayed in Table 4. Generally, all of the 
skills were rated as either essf.xtial or desirable by the 
respondents, thus indicating that the list of skills derived from 
the literature was "on target". Only advanced computer 
programming, time series design, observational research, and 
grantsmanship were rated as "not important" by more than 20 
percent of the respondents. Analysis of ratings by type of 
organization revealed a very consistent response pattern for all 
of the skill areas except in the psychometrics area. In this 
area, the rating "not important" was used by representatives of 
all of the types of organizations fixc^pt one residential school 
and all of the other educational agencies. Thus, psychometrics 
appears to be a skill area that is emphasized more in educational 
settings than in the other work settings. 

Within each work setting and in each discipline, particular 
skills will receive greater emphasis. For example, Davis (1986b) 
pointed out that psychological evaluators would emphasize goal 
attainment scaling and management information systems. 
Educational evaluators would emphasize curriculum development and 
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school organization, and business evaluators would emphasize 
organizational change, market research, and cost benefit 
analysis. While recognizing these discipline specific skill 
areas, the skill ratings in the present study indicate that there 
is a generic set of skills that apply across types of 
organizations. These include skills in traditional research 
methodology, evaluation, administration/communication, 

statistical analysis, and computer usage. 

In a paper entitled "Content Specialization and Educational 
Evaluation: A Necessary Marriage?", Worthen and Sanders (1984) 
examined the issue: Should an educational evaluator be trained 
as a content specialist, an evaluation specialist, or some 
combination of the two'. They concluded that evaluation 
specialists are the best choice to evaluate most educational 
enterprises. They recognized that content specialization plays 
an important role in educational evaluation, but it is neither 
necessary nor desirable in the training of educational 
evaluators . 

The majority of the respondents (59 percent) agreed with 
Worthen and Sanders conclusion. They felt that general research 
and evaluation skills would be sufficient and that the person 
could "learn as they go". Fifteen percent of the respondents 
felt that some expertise would be necessary in such areas as 
administration, sign lanouage, audiology, sociology, counseling, 
the law, economics, agriculture, criminal justice, mathematics, 
or reading. These respondents also felt that only some of the 
staff would need to be content specialists, one respondent 
recognized that the area of needed expertise would change based 
on what was being funded that year. The reir .ining respondents 
(21 percent) felt that expertise would be needed in such areas as 
engineering, military sciences, government documents, foreign 
languages, reading, mathematics, deafness, sign language, 
sociology, health, the law, and finance. 

The respondents were asked to add any additional skills that 
they thought would be needed in their particular setting to 
conduct evaluation and research studies. Over 60 different skill 
areas were listed; only a few of which were mentioned more than 
once. The residential schools • and two of the other educational 
agencies' representatives stressed that manual communication and 
knowledge of deafness-related issues would be important. Four of 
the educational representatives felt that knowledge of the 
subject area such as reading, mathematics, science, or social 
studies would be important. In addition, four of the educational 
representatives reported that experience in an educational 
setting would be important. 

other areas that were mentioned more than two or three times 
included: legal Issues, health, economics, personnel, finance, 
agriculture, French, Spanish, human relations skills, 
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linguistics, counseling, sociology, human development, 
psychology, audiology, and visual impairments. 

Interest in an Internship 

Sixty-two of the 69 respondents reported that they would be 
interested in having an intern work in their organisation. The 
same percentage reported an interest in having an intern who was 
hearing- impaired. one representative expressed some reservation 
about having a hearing impaired intern because much of their work 
is international and requires fluency in a foreign language. 
However, this representative said that at the entry level there 
would be appropriate positions for individuals with evaluation 
research skills that did not recjuire communication in a foreign 
langaage. She did indicate that she thought it would be hard to 
advance at that organization without these skills. 

Several important points came out of discussions about the 
internship process. First, the respondents revealed a very 
positive attitude toward having a hearing impaired intern. 
Sample comments included: 

••We try to do things better than other people. I know that a lot 
of deaf people can do a lot on their own.»» 

••We would be eager to do that (hire a hearing impaired intern) . 
We need indepth study of methods such as sampling, aggregating 
data. . .•• 

••We pride ourselves on being eclectic — being atole to work with 
diverse types of people. •• 

The idea of having an intern who was knowledgeable about deafness 
and research and evaluation appealed to the respondents from the 
residential schools for the deaf and from various social service 
agencies. Sample comments included: 

••Send •em up here for practical experience. We really do need 
someone. •• 

••Deafness doesn^t matter. Depends on the nature of the task.^^ 

••We work with hearing impaired clients, so it would be a 
natural. •• 

••More than one would be welcome. •• 

Second, the respondents emphasized the need for prior 
planning and sufficient support resources, as exemplified by the 
following comments: 

••Hearing status is not important. Availability of an internship 
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is dependent on the match between the project and the 
qualifications of the student." 

••We would be interested in hiring a hearing impaired intern. I 
am concerned about communication and not giving the intern a 
passive role for six months." 

"You would need to interview the consultants on staff to 
determine an appropriate project for them." 

"I am concerned about communication and not giving the intern a 
passive role. You need to allow nine months to formally 
establish an internship with personnel and get a sponsor to help 
you push through the paperwork." 



"We are very enthusiastic about hiring an intern. We may work 
out room and board for 20 hours of work a week." 

These comments raise several important issues. First, the 
respondents recognized the communication problems faced by the 
hearing impaired intern. Except for the residential schools, the 
majority of the respondents reported that Gallaudet would need to 
furnish an interpreter to facilitate communication. (The one 
oral residential school respondent commented that, " They must be 
able to communicate orally and must promise never to use any sign 
language.") Second, the respondents emphasized the need to find 
a match between the organization's work and the skills of the 
intern. This requires considerable planning and supervision of 
the internship placements to insure that the experience is 
meaningful to the student and that the student is making a 
meaningful contribution to the organisation. Daudistel and 
Hedderson (1984) recommend that the faculty visit the internship 
site to explain to the agency that the trainees must do research 
and/or evaluation work, not clerical tasks. The interns need to 
obtain practical experience in applying research and evaluation 
skills; they need to collect useful data in the context of an 
operating program without disrupting normal operations of the 
program. Third, the respondents seemed eager to learn about 
working with a hearing impaired person. Therefore, some deaf 
awareness education might be needed to facilitate the process. 
Fourth, the issue of paid versus non-paid inrernships needs to be 
worked out. Possible source of stipends need to be explored. 
Fifth, considerable pre-planning time is needed. 

Future Job Prospects 

In the initial screening questionnaire, respondents were 
asked to indicate how many staff and consultants they expected to 
hire in the next three years. This must have been a difficult 
question for the respondents to answer because 44 percent left it 
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blank for the number of staff members and 73 percent left it 
blank for the nximber of consultants. of those who provided an 
answer, 59 percent indicated that they planned to hire at least 
one staff member, with an average response of 2.6 people and a 
range between 1 and 15. Although fewer respondents answered the 
consultant question, 59 percent of those who did also indicated 
that they planned to hire at least one consultant, with an 
average response of 3.6 and a range between 1 and 20. 

In the telephone interview, respondents were asked to 
describe their view as to job prospects in their own organization 
and in the field in general. Their responses provide some insight 
into why the question of future hirings was so difficult. 
Basically, the respondents indicated that future hirings were 
dependent on availability of funding, economic growth, and 
attrition. Without knowing if there would be funding cut backs 
or who might be leaving and when, it was difficult for the 
respondents to specify how many new hires wouli be needed. 

Six of the eleven representatives of professional 
associations reported that job prospects in their organization 
were "slim" to "not good". The others said that the chances were 
"fair" to "good". However, only two of the eleven professional 
association representatives thought that the job prospects in the 
field in general were "slim" or "not good". The others indicated 
that prospects would be good in larger firms, or very good 
because organizations that serve the pviblic need to be 
accountable. 

Two of the nine consulting firms/private corporations 
representatives reported that they could not say for certain what 
the job prospects were because they "fluctu^4te and are hard o 
determine". However, five of the other representatives described 
their situation positively, e.g., "growing", "very likely they 
will be hiring", "good", "a variety of needs exist", and "changes 
year to year, but looks good now". Their responses followed a 
similar pattern with regard to the field in general. 

The government agency representatives painted a similarly 
positive picture, e.g., "20 to 30 jobs a year", "need to expand", 
"continue to be important to 'tell their story*", "high demand 
now", and "will be opportunities". They also felt very "bullish" 
on job prospects in the field in general, seeing "more demand for 
skills to support programs". 

The representatives of the residential schools were about 
equally split in describing the job prospects as good and poor. 
Most agreed that work needed to be done, but they wondered where 
the funds would come from. The one's who described the prospect 
as good based that on "an emphasis on long term planning", "need 
to document the job they are doing", "providing services 
throughout the state", and "good on a consultant basis". The 
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respondents that thought job prospects were poor commented that 
it was basically a problem of "a strong need but little money". 
Some did not see jobs for the ''straight evaluator/researcher at 
the individual school level, but rather at the division or state 
level". Thus, the residential school representatives generally 
saw a more positive picture for job prospects in the field in 
general than in their individual schools. 

In other educational agencies, six of the eleven 
representative described job prospects positively, e.g,, "expect 
to hire 3 to 10 people in the next 3 to 5 years", "moderately 
positive", "chances are good with research skills". Those with a 
more negative outlook again blamed the problem on inadequate 
funding. Seven of the respondents indicated a positive feeling 
about job prospects in the field in general, with one person 
predicting "a substantial increase in the need for competent 
people in evaluation and research". 

In the telephone interview, respondents were also asked to 
indicate whether or not they would hire a hearing impaired person 
at their organization to do research and evaluation. 
Interestingly, only 3 of the 69 representatives reported that 
they would not consider hiring a hearing impaired person. One 
person reported that he did not have enough business to keep 
himself busy not to mention an intern also. Another respondent 
said that their office was too small to accommodate an intern and 
an interpreter also. The third one said that they already used 
law interns and that was sufficient personnel. 

The respondents seemed to be more concerned with getting a 
qualified employee than with the person's hearing status. 
Several of the respondents raised issues similar to those 
discussed in relation to hiring a hearing impaired intern. They 
were concerned about the communication problems and who would 
provide an interpreter when it was necessary. Several indicated 
that they were "eager to hire" handicapped workers and that they 
had had hearing impaired people work for them before. 

Summary and Conclusions 

The purpose of this study was to determine the views of 
employers from a variety of job placement ^iettings and 
disciplines concerning the critical skills for the research and 
evaluation professional, implications for training, and future 
job prospects. Through a phone survey of 69 employers of 
evaluation research personnel and a mail survey of a subset of 48 
of those employers, 4n'o^^tion was obtained regarding the nature 
of the organization, the kindfi of skills needed in research and 
evaluation, the respondent's background, interest in beina an 
internship site, and future job prospects. The organizations 
surveyed represented consulting firms, private corporations, 
government agencies, residential schools for the deaf, other 
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educational agencies, and professional associations. 

The results of the survey support the position that it is 
possible to identify a generic set of skills needed to do 
research and evaluation that apply across a variety of types of 
organizations. These include skills in traditional research 
methodology, evaluation, administration/comiaunication, 

statistical analysis, and computer usage. in some settings, 
particular skills would be emphasized more than in others. For 
example, in residential schools, knowledge of deafness and sign 
languaae vexcept at an oral school) would be important, in some 
educational settings, knowledae of psychometrics or specific 
subject areas such as mathematics or reading would be important, 
other organizations reported that it would be desirable to have 
expertise in such areas as social welfare, law, criminal justice, 
foreign languages, military, health, finance, personnel, and 
human relations skills. An interdisciplinary approach to 
teaching evaluation research would focus on the generic skills 
and allow enough flexibility to address these specific skill 
areas as well. 

Sixty-two of the 69 respondents reported that they would be 
interested in having an intern work in their organization, 
whether the intern was normally hearing or hearing impaired. 
Their comments revealed a very positive attitude toward having a 
hearing impaired intern. The idea of having an intern who was 
knowledgeable about deafness and research and evaluation appealed 
to the respondents from the residential schools for the deaf and 
from various social service agencies. The respondents also 
stressed the need for prior planning and sufficient support 
services (in the way of interpreters) to insure that the 
internship experience is meaningful to the student and that the 
student is making a meaningful contribution to the organization. 
Education in deaf awareness may be needed in some organizations. 
The issue of paid versus non-paid internships needs to be worked 
out and possible sources of stipends need to be explored. 

Projecting job openings is a difficult task, given that such 
openings are dependent on the availability of funds, economic 
growth, and attrition. However, even with this uncertainty, 59 
percent of the respondents who were willing to speculate on this 
reported that they expected to be hiring evaluation research 
personnel in the next three years. Respondents in smaller 
professional associations were not optimistic about future job 
prospects. However, those in larger firms saw an increased need 
to be accountable to their public and thus were more optimistic 
in their outlook. Representative of the consulting firms, 
private corporations, and government agencies were generally 
positive with regard to future job prospects, e.g., "chamges year 
to year, but looks good now", "20 to 30 jobs a year", "need to 
expand", and "more demand for skills to support programs". The 
representatives from the residential schools were mixed in their 
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opinions of future job prospects. They agreed that the work 
needed to be done, but wondered where the funds would come from. 
Job prospects were more positive in schools with an emphasis on 
planning and documentation of their services. m the other 
educational agencies, the picture was also mixed and was also 
blamed on the problem of inadequate funds. Seven of the eleven 
respondents from the educational agencies indicated a positive 
feeling about job prospects in the field in general. Overall, 
the results indicate a very receptive job market for both 
normally hearing and hearing impaired evaluators/ researchers or 
for evaluators/researchers in the area of deafness. 

Are the results biased because the agency conducting the 
study educates both normally hearing and hearing impaired 
students at the graduate level? The influence of such a bias 
might be demonstrated in a more restricted range of skills that 
employers thought a hearing impaired person could manage, or in 
more pessimistic projections regarding internships and job 
prospects. However, the list of skills that resulted correspond 
fairly well to those reported by other researchers (e.g., Davis, 
1986b; Anderson & Ball, i979; Maryak, Gray, Mehrens, & Lezotte, 
1979). The prospects for atemshlps and job placements were 
both very positive. This result may be more positive than would 
be found in other regions of the country because the study 
focused on the greater Washington, D.C. area. The multitude of 
government agencies, professional associations, and consulting 
firms provide a more hospitable climate for the research and 
evaluation professional. The results concerning internships and 
job placements might be replicated in other major metropolitan 
areas with a similar economic structure. 
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GALLAUDET COLLEGE 

RESPONSE CARD 



Company Hnstitution: 

Contact Person: 

Telephone Numbcri_ 



Number and type of individuals who use evaluation /research skills: 



LEVEL 


ON-STAFF 


ON CONSULTANT BASIS 


Number 


Field of Expertise 


Number 


Field of Fxpcnise 


Doctoral 










Master's 










Bachelor's 










Hired in '84 










Expect to hire in 
the next 3 vcars 











Sample fields of expertise: evaluation, education, business, statistics, computer programming, 
political science, public administration, psychology, anthropology, sociology, economics, other 
(please specify). 



Would you be willing to participate in a 20-minute phone conversation to help us identify specific evaluation 
research skills that you look for in individuals you hire^ □ Yes □ No 

Please fold the response card so that the return address is on the outside. Staple or tape dosed. 



THANK YOU! 



ERIC 



24 



Please indicate the importance ol" the following skills for an evaluation research professional by 
checking the appropriate box using the following scale: 



Essential 



2 — Desirable 



3 — Not important 



RESEARCH METHODS 







Nol 






Es«eniial 


Oninbic 


Important 






1 

□ 


2 

□ 


i 

□ 


1. 


Sampling 


n 


n 


□ 


2 


Survey research 


□ 


□ 


□ 


3 


Questionnaire construction 


□ 


□ 


□ 


4. 


Interviewing skills 


□ 


□ 


□ 


5. 


Content analysis 


□ 


□ 


□ 


6. 


Case studies 


□ 


□ 


□ 


7. 


Task analysis 


□ 


□ 


□ 


8. 


Experimental design 


□ 


□ 


□ 


9 


Quasi-experimental design 


□ 


□ 


□ 


10. 


Time series design 


□ 


□ 


□ 


11. 


Observational research 



EVALUATION 



□ 


□ 


□ 


1. 


Alternative models for evaluation 


□ 


□ 


□ 


2. 


Setting goals 


□ 


□ 


□ 


3. 


Cost-benefit analysis 


□ 


□ 


□ 


4. 


Grantsmanship 


□ 


□ 


□ 


5. 


Standards for evaluation (ethical, legal) 


□ 


□ 


□ 


6 


Utilization of evaluation results 


□ 


□ 


□ 


7. 


Planning evaluations 


□ 


□ 


□ 


8. 


Assessment of program implementation 


□ 


□ 


□ 


9 


Meta-evaluation 


□ 


□ 


□ 


10. 


Politics of evaluation 


□ 


□ 


□ 


11. 


Policy analysis 


□ 


□ 


□ 


12. 


Needs assessment 



STATISTICS /COMPUTER 







Not 






Essential 


DesirabU 


Imporiani 






1 

□ 


2 

□ 


3 

□ 


I. 


Data preparation 


□ 


□ 


n 


2. 


Construction of data bases 


□ 


□ 


□ 


3. 


Descriptive statistics 


□ 


□ 


□ 


4. 


Inferential statistics 


□ 


□ 


□ 


5. 


Correlation and regression 


□ 


□ 


□ 


6. 


Multivariate statistics 


□ 


□ 


□ 


7. 


Nonparametric statistics 


□ 


□ 


□ 


8 


Quality control 


□ 


□ 


□ 


9 


Elementary coKiputer programming 


□ 


□ 


□ 


10. 


Advanced computer programming 


□ 


□ 


□ 


11. 


Use of canned programs 


□ 


□ 


□ 


12 


Use of micro computer software 



PSYCHOMETRICS 



□ 


□ 


□ 


1 


Test construction 


□ 


□ 


□ 


2 


Reliability 


□ 


□ 


□ 


3. 


Validity 


□ 


□ 


□ 


4. 


Application of tests (paper and pencil; situational 










performance) 


□ 


□ 


□ 


5 


Norm and criterion — referenced tests 


□ 


□ 


□ 


6. 


Selecting a measurement instrument 


□ 


□ 


□ 


7 


Assessing a measurement instrument 


□ 


□ 


□ 


8. 


interpreting test results 


□ 


□ 


□ 


9. 


Reactive concerns and unobstrusive measures 



ADMINISTRATION / COMMUNICATION 
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□ 


□ 


□ 


1. 


Proposal writing 


□ 


□ 


□ 


2. 


Expository skills (writing and speaking) 


□ 


□ 


□ 


3. 


Public relations skills 


□ 


□ 


□ 


4. 


Management skills 


n 


□ 


□ 


5. 


Technical writing of reports 


□ 


□ 


□ 


6. 


Listening skills 


□ 


□ 


□ 


7. 


Consultation skills 



Please fold this form so that the return address ts on the outside . 
Staple or tape closed. 

THANK YOU! 
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INTERVIEW GUIDE FOR RESPONDENTS 



This is lYOUR^NRMEl . 

(Ask to speak to the contact person. If they are not available, 
ask for a good time to call back. Note that time in pencil on 
this sheet. ) 



(If you cire able to talk to the contact person:) 

I am calling from Gallaudet College concerning an evaluation 
research program that we are making plans to develop. I believe 
you received a questionnaire in the mail from us. 

BRPNCH 1 

YES PEOPLE 

(If the person responded to the first questionnaire and said that 
he/she would be willing to talk with us, say:) 

You responded to our first questionnaire and indicated that you 
would be willing to talk with us about this a little bit more. 
Is now a good time to talk? 

(If not, schedule a convenient time to call back and make a note 
of it.) (If yes, go to l.l). 



BLANK PEOPLE 

(If the person responded to the first questionnaire and left 
blank the indication of being willing to talk with us, sa;^ .) 
You responded to our first questionnaire and I was wondering if 
you would be willing to talk to me a bit more? about it now? 
(If not , say : ) 

What would be a more convenient time to call back? (Make a note). 



1. 1 BEGIN INTERVIEW 

I want to minimize the amount of time that I i.ake from you and 
maximize the outcome of our conversation, so to begin with, I 
want tc briefly share with you our ideas of who an 
evaluat or/ researcher is and what such a person does. 

We are operating from the definition of an evaluator as a person 
who can help an organization clarify its goals and determine how 
well the organization is meeting those goals. An 
evaluator/ researcher uses the scientific method to collect and 
analyze data that is used to help clarify goals and to assess the 
extent to which a program or activity achieves its specified 
objectives. The evaluator/reBearcher' s skills can be 
applied in schools in terms of evaluating individual needs and 
r — (5 — ''Wd^^ce as well as at the program level. They can also be 
«pD!(^>d in market research or in establ istHwnt cf new programs or 





in projects involving survey research. 

Could you give me an idea of the kind of program/activity your 
organization is involved in? 



You say you have lBiEh_tbiS_!iy!!!kg]2_i!E_f I22?!!-.tbS_f iC^t 

guest i,onnai.rel individuals who do this kind of work- Could you 

describe for me the kind of work that these individuals do? 



Mould you give me one or two specific examples of 
evaluation/research projects/activities that are done a your 
organ izat ion? 



UouiH it be important fcr an eval uat or/researcher to be a content 
specialist in your organization? 

What content areas of specialization would you be looking for 
(e* g. , reading, economics, health, criminal justice, 
parent — ch i 1 d ) ? 



I sent you a list of skills that might be important for an 
eval uat or /researcher. Do you have that list in front of you? 

Please take a moment to scan these. Ore there any skills not 
listed that you think should be? 



Would you complete the rating scale for the skills and mail it to 
me? filso, if you don't mind, would you write (the_name_of _the 
ECCBSDM^ that we won't need to bother you again? 

Thanks. 

ft few last questions. ftre you yourself trained in 
evaluation/research? What was the nature of your training? 



ftre you doing eval uat ion /research yourself? How many ysars have 
you been doing work in this area? 




2» 



Ue think that practical experience is an important part of 
becoming a skilled eval uator/researcher. Ue plan to first teach 
the necessary skills to our students and then provide them with 
an internship to obtain this practical experience. 

2.1 If the logistics could be worked out in an acceptable way, 
(i.e., supervision from a faculty member, timeliness) do you 
think there would be interest in having an intern from our 
program work iri your organization? (IF YES TO 2.1, GO TO 2.2. 
IF NO ro 2.1, GO TO 2.4). 

BRANCH 2 

2.2 (IF YES TO 2. 1 : ) 

Could we come visit your organization and talk with you in more 
detail about what those logistics would be? 

2.3 (IF YES to 2.2, schedule a time to visit). STOP. THONK 



(IF NO to 2.2, ask:) 

What would the process be for establishing an internship? 
Who would we need to contact? 



Some of our students might be hearing impaired. fts you know, 
some hearing impaired people need an interpreter to function in 
the "hearing" world. Would your organization be willing to 
consider hiring an hearing impaired intern? 



If we provided the interpreter? 



2. f> What do you see as future job prospects for 

evaluation/research persons in the future at your organization? 



General ly? 



Would your organization be willing to consider hiring a hearing 
impaired graduate of our program? 
If an interpreter was provided? 



YOU. 





THfiNK YOU VERY MUCH. 



£.4 (IF NO TO 2. 1, SfiY:> 

What do you see as future Job prospects for evaluation/research 
persons in the future at your organization? Generally? 



Some of our students might Le hearing impaired. fts you know, 
some hearing impaired people need an interpreter to function in 
the "hearing" world. Would your organization be willing to 
consider hiring ^ hearing impaired graduate of our program? 
If an interpreter was provided? 



THONK YQU VERY MUCH. 



3u 



